Improvise a Jazz Solo with an LSTM Network

Welcome to your final programming assignment of this week! In this notebook, you will implement a model that uses an LSTM to generate music. At the end, you'll even be able to listen to your own music!

By the end of this assignment, you'll be able to:

  • Apply an LSTM to a music generation task
  • Generate your own jazz music with deep learning
  • Use the flexible Functional API to create complex models

This is going to be a fun one. Let's get started!

Packages

Run the following cell to load all the packages you'll need. This may take a few minutes!

In [4]:
import IPython
import sys
import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf

from music21 import *
from grammar import *
from qa import *
from preprocess import * 
from music_utils import *
from data_utils import *
from outputs import *
from test_utils import *

from tensorflow.keras.layers import Dense, Activation, Dropout, Input, LSTM, Reshape, Lambda, RepeatVector
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.utils import to_categorical

1 - Problem Statement

You would like to create a jazz music piece specially for a friend's birthday. However, you don't know how to play any instruments, or how to compose music. Fortunately, you know deep learning and will solve this problem using an LSTM network!

You will train a network to generate novel jazz solos in a style representative of a body of performed work. 😎🎷

1.1 - Dataset

To get started, you'll train your algorithm on a corpus of Jazz music. Run the cell below to listen to a snippet of the audio from the training set:

In [5]:
IPython.display.Audio('./data/30s_seq.wav')
Out[5]:

The preprocessing of the musical data has been taken care of already, which for this notebook means it's been rendered in terms of musical "values."

What are musical "values"? (optional)

You can informally think of each "value" as a note, which comprises a pitch and duration. For example, if you press down a specific piano key for 0.5 seconds, then you have just played a note. In music theory, a "value" is actually more complicated than this -- specifically, it also captures the information needed to play multiple notes at the same time. For example, when playing a music piece, you might press down two piano keys at the same time (playing multiple notes at the same time generates what's called a "chord"). But you don't need to worry about the details of music theory for this assignment.

Music as a sequence of values

  • For the purposes of this assignment, all you need to know is that you'll obtain a dataset of values, and will use an RNN model to generate sequences of values.
  • Your music generation system will use 90 unique values.

Run the following code to load the raw music data and preprocess it into values. This might take a few minutes!

In [6]:
X, Y, n_values, indices_values, chords = load_music_utils('data/original_metheny.mid')
print('number of training examples:', X.shape[0])
print('Tx (length of sequence):', X.shape[1])
print('total # of unique values:', n_values)
print('shape of X:', X.shape)
print('Shape of Y:', Y.shape)
print('Number of chords', len(chords))
number of training examples: 60
Tx (length of sequence): 30
total # of unique values: 90
shape of X: (60, 30, 90)
Shape of Y: (30, 60, 90)
Number of chords 19

You have just loaded the following:

  • X: This is an (m, $T_x$, 90) dimensional array.

    • You have m training examples, each of which is a snippet of $T_x =30$ musical values.
    • At each time step, the input is one of 90 different possible values, represented as a one-hot vector.
      • For example, X[i,t,:] is a one-hot vector representing the value of the i-th example at time t.
  • Y: a $(T_y, m, 90)$ dimensional array

    • This is essentially the same as X, but shifted one step to the left (to the past).
    • Notice that the data in Y is reordered to be dimension $(T_y, m, 90)$, where $T_y = T_x$. This format makes it more convenient to feed into the LSTM later.
    • Similar to the dinosaur assignment, you're using the previous values to predict the next value.
      • So your sequence model will try to predict $y^{\langle t \rangle}$ given $x^{\langle 1\rangle}, \ldots, x^{\langle t \rangle}$.
  • n_values: The number of unique values in this dataset. This should be 90.

  • indices_values: python dictionary mapping integers 0 through 89 to musical values.

  • chords: Chords used in the input midi

1.2 - Model Overview

Here is the architecture of the model you'll use. It's similar to the Dinosaurus model, except that you'll implement it in Keras.

Figure 1: Basic LSTM model
  • $X = (x^{\langle 1 \rangle}, x^{\langle 2 \rangle}, \cdots, x^{\langle T_x \rangle})$ is a window of size $T_x$ scanned over the musical corpus.
  • Each $x^{\langle t \rangle}$ is an index corresponding to a value.
  • $\hat{y}^{\langle t \rangle}$ is the prediction for the next value.
  • You'll be training the model on random snippets of 30 values taken from a much longer piece of music.
    • Thus, you won't bother to set the first input $x^{\langle 1 \rangle} = \vec{0}$, since most of these snippets of audio start somewhere in the middle of a piece of music.
    • You're setting each of the snippets to have the same length $T_x = 30$ to make vectorization easier.

Overview of Section 2 and 3

In Section 2, you're going to train a model that predicts the next note in a style similar to the jazz music it's trained on. The training is contained in the weights and biases of the model.

Then, in Section 3, you're going to use those weights and biases in a new model that predicts a series of notes, and using the previous note to predict the next note.

  • The weights and biases are transferred to the new model using the global shared layers (LSTM_cell, densor, reshaper) described below

2 - Building the Model

Now, you'll build and train a model that will learn musical patterns.

  • The model takes input X of shape $(m, T_x, 90)$ and labels Y of shape $(T_y, m, 90)$.
  • You'll use an LSTM with hidden states that have $n_{a} = 64$ dimensions.
In [7]:
# number of dimensions for the hidden state of each LSTM cell.
n_a = 64 

Sequence generation uses a for-loop

  • If you're building an RNN where, at test time, the entire input sequence $x^{\langle 1 \rangle}, x^{\langle 2 \rangle}, \ldots, x^{\langle T_x \rangle}$ is given in advance, then Keras has simple built-in functions to build the model.
  • However, for sequence generation, at test time you won't know all the values of $x^{\langle t\rangle}$ in advance.
  • Instead, you'll generate them one at a time using $x^{\langle t\rangle} = y^{\langle t-1 \rangle}$.
    • The input at time "t" is the prediction at the previous time step "t-1".
  • So you'll need to implement your own for-loop to iterate over the time steps.

Shareable weights

  • The function djmodel() will call the LSTM layer $T_x$ times using a for-loop.
  • It is important that all $T_x$ copies have the same weights.
    • The $T_x$ steps should have shared weights that aren't re-initialized.
  • Referencing a globally defined shared layer will utilize the same layer-object instance at each time step.
  • The key steps for implementing layers with shareable weights in Keras are:
  1. Define the layer objects (you'll use global variables for this).
  2. Call these objects when propagating the input.

3 types of layers

  • The layer objects you need for global variables have been defined.
    • Just run the next cell to create them!
  • Please read the Keras documentation and understand these layers:
    • Reshape(): Reshapes an output to a certain shape.
    • LSTM(): Long Short-Term Memory layer
    • Dense(): A regular fully-connected neural network layer.
In [8]:
n_values = 90 # number of music values
reshaper = Reshape((1, n_values))                  # Used in Step 2.B of djmodel(), below
LSTM_cell = LSTM(n_a, return_state = True)         # Used in Step 2.C
densor = Dense(n_values, activation='softmax')     # Used in Step 2.D
  • reshaper, LSTM_cell and densor are globally defined layer objects that you'll use to implement djmodel().
  • In order to propagate a Keras tensor object X through one of these layers, use layer_object().
    • For one input, use layer_object(X)
    • For more than one input, put the inputs in a list: layer_object([X1,X2])

Exercise 1 - djmodel

Implement djmodel().

Inputs (given)

  • The Input() layer is used for defining the input X as well as the initial hidden state 'a0' and cell state c0.
  • The shape parameter takes a tuple that does not include the batch dimension (m).
    • For example,
      X = Input(shape=(Tx, n_values)) # X has 3 dimensions and not 2: (m, Tx, n_values)
      

Step 1: Outputs

  • Create an empty list "outputs" to save the outputs of the LSTM Cell at every time step.

Step 2: Loop through time steps

  • Loop for $t \in 1, \ldots, T_x$:

2A. Select the 't' time-step vector from X.

  • X has the shape (m, Tx, n_values).
  • The shape of the 't' selection should be (n_values,).
  • Recall that if you were implementing in numpy instead of Keras, you would extract a slice from a 3D numpy array like this:
    var1 = array1[:,1,:]
    

2B. Reshape x to be (1, n_values).

  • Use the reshaper() layer. This is a function that takes the previous layer as its input argument.

2C. Run x through one step of LSTM_cell.

  • Initialize the LSTM_cell with the previous step's hidden state $a$ and cell state $c$.
  • Use the following formatting:
    next_hidden_state, _, next_cell_state = LSTM_cell(inputs=input_x, initial_state=[previous_hidden_state, previous_cell_state])
    
    • Choose appropriate variables for inputs, hidden state and cell state.

2D. Dense layer

  • Propagate the LSTM's hidden state through a dense+softmax layer using densor.

2E. Append output

  • Append the output to the list of "outputs".

Step 3: After the loop, create the model

  • Use the Keras Model object to create a model. There are two ways to instantiate the Model object. One is by subclassing, which you won't use here. Instead, you'll use the highly flexible Functional API, which you may remember from an earlier assignment in this course! With the Functional API, you'll start from your Input, then specify the model's forward pass with chained layer calls, and finally create the model from inputs and outputs.

  • Specify the inputs and output like so:

    model = Model(inputs=[input_x, initial_hidden_state, initial_cell_state], outputs=the_outputs)
    
    • Then, choose the appropriate variables for the input tensor, hidden state, cell state, and output.
  • See the documentation for Model
In [9]:
# UNQ_C1 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: djmodel

def djmodel(Tx, LSTM_cell, densor, reshaper):
    """
    Implement the djmodel composed of Tx LSTM cells where each cell is responsible
    for learning the following note based on the previous note and context.
    Each cell has the following schema: 
            [X_{t}, a_{t-1}, c0_{t-1}] -> RESHAPE() -> LSTM() -> DENSE()
    Arguments:
        Tx -- length of the sequences in the corpus
        LSTM_cell -- LSTM layer instance
        densor -- Dense layer instance
        reshaper -- Reshape layer instance
    
    Returns:
        model -- a keras instance model with inputs [X, a0, c0]
    """
    # Get the shape of input values
    n_values = densor.units
    
    # Get the number of the hidden state vector
    n_a = LSTM_cell.units
    
    # Define the input layer and specify the shape
    X = Input(shape=(Tx, n_values)) 
    
    # Define the initial hidden state a0 and initial cell state c0
    # using `Input`
    a0 = Input(shape=(n_a,), name='a0')
    c0 = Input(shape=(n_a,), name='c0')
    a = a0
    c = c0
    ### START CODE HERE ### 
    # Step 1: Create empty list to append the outputs while you iterate (≈1 line)
    outputs = []
    
    # Step 2: Loop over tx
    for t in range(Tx):
        
        # Step 2.A: select the "t"th time step vector from X. 
        x = X[:,t,:]
        # Step 2.B: Use reshaper to reshape x to be (1, n_values) (≈1 line)
        x = reshaper(x)
        # Step 2.C: Perform one step of the LSTM_cell
        a, _, c = LSTM_cell(inputs=x, initial_state=[a, c])
        # Step 2.D: Apply densor to the hidden state output of LSTM_Cell
        out = densor(a)
        # Step 2.E: add the output to "outputs"
        outputs.append(out)
        
    # Step 3: Create model instance
    model = Model(inputs=[X, a0, c0], outputs=outputs)
    
    ### END CODE HERE ###
    
    return model

Create the model object

  • Run the following cell to define your model.
  • We will use Tx=30.
  • This cell may take a few seconds to run.
In [10]:
model = djmodel(Tx=30, LSTM_cell=LSTM_cell, densor=densor, reshaper=reshaper)
In [11]:
# UNIT TEST
output = summary(model) 
comparator(output, djmodel_out)
All tests passed!
In [12]:
# Check your model
model.summary()
Model: "functional_1"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_1 (InputLayer)            [(None, 30, 90)]     0                                            
__________________________________________________________________________________________________
tf_op_layer_strided_slice (Tens [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
reshape (Reshape)               (None, 1, 90)        0           tf_op_layer_strided_slice[0][0]  
                                                                 tf_op_layer_strided_slice_1[0][0]
                                                                 tf_op_layer_strided_slice_2[0][0]
                                                                 tf_op_layer_strided_slice_3[0][0]
                                                                 tf_op_layer_strided_slice_4[0][0]
                                                                 tf_op_layer_strided_slice_5[0][0]
                                                                 tf_op_layer_strided_slice_6[0][0]
                                                                 tf_op_layer_strided_slice_7[0][0]
                                                                 tf_op_layer_strided_slice_8[0][0]
                                                                 tf_op_layer_strided_slice_9[0][0]
                                                                 tf_op_layer_strided_slice_10[0][0
                                                                 tf_op_layer_strided_slice_11[0][0
                                                                 tf_op_layer_strided_slice_12[0][0
                                                                 tf_op_layer_strided_slice_13[0][0
                                                                 tf_op_layer_strided_slice_14[0][0
                                                                 tf_op_layer_strided_slice_15[0][0
                                                                 tf_op_layer_strided_slice_16[0][0
                                                                 tf_op_layer_strided_slice_17[0][0
                                                                 tf_op_layer_strided_slice_18[0][0
                                                                 tf_op_layer_strided_slice_19[0][0
                                                                 tf_op_layer_strided_slice_20[0][0
                                                                 tf_op_layer_strided_slice_21[0][0
                                                                 tf_op_layer_strided_slice_22[0][0
                                                                 tf_op_layer_strided_slice_23[0][0
                                                                 tf_op_layer_strided_slice_24[0][0
                                                                 tf_op_layer_strided_slice_25[0][0
                                                                 tf_op_layer_strided_slice_26[0][0
                                                                 tf_op_layer_strided_slice_27[0][0
                                                                 tf_op_layer_strided_slice_28[0][0
                                                                 tf_op_layer_strided_slice_29[0][0
__________________________________________________________________________________________________
a0 (InputLayer)                 [(None, 64)]         0                                            
__________________________________________________________________________________________________
c0 (InputLayer)                 [(None, 64)]         0                                            
__________________________________________________________________________________________________
tf_op_layer_strided_slice_1 (Te [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
lstm (LSTM)                     [(None, 64), (None,  39680       reshape[0][0]                    
                                                                 a0[0][0]                         
                                                                 c0[0][0]                         
                                                                 reshape[1][0]                    
                                                                 lstm[0][0]                       
                                                                 lstm[0][2]                       
                                                                 reshape[2][0]                    
                                                                 lstm[1][0]                       
                                                                 lstm[1][2]                       
                                                                 reshape[3][0]                    
                                                                 lstm[2][0]                       
                                                                 lstm[2][2]                       
                                                                 reshape[4][0]                    
                                                                 lstm[3][0]                       
                                                                 lstm[3][2]                       
                                                                 reshape[5][0]                    
                                                                 lstm[4][0]                       
                                                                 lstm[4][2]                       
                                                                 reshape[6][0]                    
                                                                 lstm[5][0]                       
                                                                 lstm[5][2]                       
                                                                 reshape[7][0]                    
                                                                 lstm[6][0]                       
                                                                 lstm[6][2]                       
                                                                 reshape[8][0]                    
                                                                 lstm[7][0]                       
                                                                 lstm[7][2]                       
                                                                 reshape[9][0]                    
                                                                 lstm[8][0]                       
                                                                 lstm[8][2]                       
                                                                 reshape[10][0]                   
                                                                 lstm[9][0]                       
                                                                 lstm[9][2]                       
                                                                 reshape[11][0]                   
                                                                 lstm[10][0]                      
                                                                 lstm[10][2]                      
                                                                 reshape[12][0]                   
                                                                 lstm[11][0]                      
                                                                 lstm[11][2]                      
                                                                 reshape[13][0]                   
                                                                 lstm[12][0]                      
                                                                 lstm[12][2]                      
                                                                 reshape[14][0]                   
                                                                 lstm[13][0]                      
                                                                 lstm[13][2]                      
                                                                 reshape[15][0]                   
                                                                 lstm[14][0]                      
                                                                 lstm[14][2]                      
                                                                 reshape[16][0]                   
                                                                 lstm[15][0]                      
                                                                 lstm[15][2]                      
                                                                 reshape[17][0]                   
                                                                 lstm[16][0]                      
                                                                 lstm[16][2]                      
                                                                 reshape[18][0]                   
                                                                 lstm[17][0]                      
                                                                 lstm[17][2]                      
                                                                 reshape[19][0]                   
                                                                 lstm[18][0]                      
                                                                 lstm[18][2]                      
                                                                 reshape[20][0]                   
                                                                 lstm[19][0]                      
                                                                 lstm[19][2]                      
                                                                 reshape[21][0]                   
                                                                 lstm[20][0]                      
                                                                 lstm[20][2]                      
                                                                 reshape[22][0]                   
                                                                 lstm[21][0]                      
                                                                 lstm[21][2]                      
                                                                 reshape[23][0]                   
                                                                 lstm[22][0]                      
                                                                 lstm[22][2]                      
                                                                 reshape[24][0]                   
                                                                 lstm[23][0]                      
                                                                 lstm[23][2]                      
                                                                 reshape[25][0]                   
                                                                 lstm[24][0]                      
                                                                 lstm[24][2]                      
                                                                 reshape[26][0]                   
                                                                 lstm[25][0]                      
                                                                 lstm[25][2]                      
                                                                 reshape[27][0]                   
                                                                 lstm[26][0]                      
                                                                 lstm[26][2]                      
                                                                 reshape[28][0]                   
                                                                 lstm[27][0]                      
                                                                 lstm[27][2]                      
                                                                 reshape[29][0]                   
                                                                 lstm[28][0]                      
                                                                 lstm[28][2]                      
__________________________________________________________________________________________________
tf_op_layer_strided_slice_2 (Te [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
tf_op_layer_strided_slice_3 (Te [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
tf_op_layer_strided_slice_4 (Te [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
tf_op_layer_strided_slice_5 (Te [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
tf_op_layer_strided_slice_6 (Te [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
tf_op_layer_strided_slice_7 (Te [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
tf_op_layer_strided_slice_8 (Te [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
tf_op_layer_strided_slice_9 (Te [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
tf_op_layer_strided_slice_10 (T [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
tf_op_layer_strided_slice_11 (T [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
tf_op_layer_strided_slice_12 (T [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
tf_op_layer_strided_slice_13 (T [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
tf_op_layer_strided_slice_14 (T [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
tf_op_layer_strided_slice_15 (T [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
tf_op_layer_strided_slice_16 (T [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
tf_op_layer_strided_slice_17 (T [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
tf_op_layer_strided_slice_18 (T [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
tf_op_layer_strided_slice_19 (T [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
tf_op_layer_strided_slice_20 (T [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
tf_op_layer_strided_slice_21 (T [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
tf_op_layer_strided_slice_22 (T [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
tf_op_layer_strided_slice_23 (T [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
tf_op_layer_strided_slice_24 (T [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
tf_op_layer_strided_slice_25 (T [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
tf_op_layer_strided_slice_26 (T [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
tf_op_layer_strided_slice_27 (T [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
tf_op_layer_strided_slice_28 (T [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
tf_op_layer_strided_slice_29 (T [(None, 90)]         0           input_1[0][0]                    
__________________________________________________________________________________________________
dense (Dense)                   (None, 90)           5850        lstm[0][0]                       
                                                                 lstm[1][0]                       
                                                                 lstm[2][0]                       
                                                                 lstm[3][0]                       
                                                                 lstm[4][0]                       
                                                                 lstm[5][0]                       
                                                                 lstm[6][0]                       
                                                                 lstm[7][0]                       
                                                                 lstm[8][0]                       
                                                                 lstm[9][0]                       
                                                                 lstm[10][0]                      
                                                                 lstm[11][0]                      
                                                                 lstm[12][0]                      
                                                                 lstm[13][0]                      
                                                                 lstm[14][0]                      
                                                                 lstm[15][0]                      
                                                                 lstm[16][0]                      
                                                                 lstm[17][0]                      
                                                                 lstm[18][0]                      
                                                                 lstm[19][0]                      
                                                                 lstm[20][0]                      
                                                                 lstm[21][0]                      
                                                                 lstm[22][0]                      
                                                                 lstm[23][0]                      
                                                                 lstm[24][0]                      
                                                                 lstm[25][0]                      
                                                                 lstm[26][0]                      
                                                                 lstm[27][0]                      
                                                                 lstm[28][0]                      
                                                                 lstm[29][0]                      
==================================================================================================
Total params: 45,530
Trainable params: 45,530
Non-trainable params: 0
__________________________________________________________________________________________________

Expected Output
Scroll to the bottom of the output, and you'll see the following:

Total params: 45,530
Trainable params: 45,530
Non-trainable params: 0

Compile the model for training

  • You now need to compile your model to be trained.
  • We will use:
    • optimizer: Adam optimizer
    • Loss function: categorical cross-entropy (for multi-class classification)
In [41]:
opt = Adam(lr=0.01, beta_1=0.9, beta_2=0.999, decay=0.01)

model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])

Initialize hidden state and cell state

Finally, let's initialize a0 and c0 for the LSTM's initial state to be zero.

In [42]:
m = 60
a0 = np.zeros((m, n_a))
c0 = np.zeros((m, n_a))

Train the model

You're now ready to fit the model!

  • You'll turn Y into a list, since the cost function expects Y to be provided in this format.
    • list(Y) is a list with 30 items, where each of the list items is of shape (60,90).
    • Train for 100 epochs (This will take a few minutes).
In [43]:
history = model.fit([X, a0, c0], list(Y), epochs=100, verbose = 0)
In [16]:
print(f"loss at epoch 1: {history.history['loss'][0]}")
print(f"loss at epoch 100: {history.history['loss'][99]}")
plt.plot(history.history['loss'])
loss at epoch 1: 130.01553344726562
loss at epoch 100: 9.667448997497559
Out[16]:
[<matplotlib.lines.Line2D at 0x7f9308617290>]

Expected Output

The model loss will start high, (100 or so), and after 100 epochs, it should be in the single digits. These won't be the exact number that you'll see, due to random initialization of weights.
For example:

loss at epoch 1: 129.88641357421875
...

Scroll to the bottom to check Epoch 100

loss at epoch 100: 9.21483039855957

Now that you have trained a model, let's go to the final section to implement an inference algorithm, and generate some music!

3 - Generating Music

You now have a trained model which has learned the patterns of a jazz soloist. You can now use this model to synthesize new music!

3.1 - Predicting & Sampling

Figure 2: Generating new values in an LSTM </font>

At each step of sampling, you will:

  • Take as input the activation 'a' and cell state 'c' from the previous state of the LSTM.
  • Forward propagate by one step.
  • Get a new output activation, as well as cell state.
  • The new activation 'a' can then be used to generate the output using the fully connected layer, densor.

Initialization

  • You'll initialize the following to be zeros:
    • x0
    • hidden state a0
    • cell state c0

Exercise 2 - music_inference_model

Implement music_inference_model() to sample a sequence of musical values.

Here are some of the key steps you'll need to implement inside the for-loop that generates the $T_y$ output characters:

Step 1: Create an empty list "outputs" to save the outputs of the LSTM Cell at every time step.

Step 2.A: Use LSTM_Cell, which takes in the input layer, as well as the previous step's 'c' and 'a' to generate the current step's 'c' and 'a'.

next_hidden_state, _, next_cell_state = LSTM_cell(input_x, initial_state=[previous_hidden_state, previous_cell_state])
  • Choose the appropriate variables for input_x, hidden_state, and cell_state

2.B: Compute the output by applying densor to compute a softmax on 'a' to get the output for the current step.

2.C: Append the output to the list outputs.

2.D: Convert the last output into a new input for the next time step. You will do this in 2 substeps:

  • Get the index of the maximum value of the predicted output using tf.math.argmax along the last axis.
  • Convert the index into its n_values-one-hot encoding using tf.one_hot.

2.E: Use RepeatVector(1)(x) to convert the output of the one-hot enconding with shape=(None, 90) into a tensor with shape=(None, 1, 90)

Step 3: Inference Model:

This is how to use the Keras Model object:

model = Model(inputs=[input_x, initial_hidden_state, initial_cell_state], outputs=the_outputs)
  • Choose the appropriate variables for the input tensor, hidden state, cell state, and output.

Hint: the inputs to the model are the initial inputs and states.

In [62]:
# UNQ_C2 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: music_inference_model

def music_inference_model(LSTM_cell, densor, Ty=100):
    """
    Uses the trained "LSTM_cell" and "densor" from model() to generate a sequence of values.
    
    Arguments:
    LSTM_cell -- the trained "LSTM_cell" from model(), Keras layer object
    densor -- the trained "densor" from model(), Keras layer object
    Ty -- integer, number of time steps to generate
    
    Returns:
    inference_model -- Keras model instance
    """
    
    # Get the shape of input values
    n_values = densor.units
    # Get the number of the hidden state vector
    n_a = LSTM_cell.units
    
    # Define the input of your model with a shape 
    x0 = Input(shape=(1, n_values))
    
    
    # Define s0, initial hidden state for the decoder LSTM
    a0 = Input(shape=(n_a,), name='a0')
    c0 = Input(shape=(n_a,), name='c0')
    a = a0
    c = c0
    x = x0

    ### START CODE HERE ###
    # Step 1: Create an empty list of "outputs" to later store your predicted values (≈1 line)
    outputs = []
    
    # Step 2: Loop over Ty and generate a value at every time step
    for t in range(Ty):
        # Step 2.A: Perform one step of LSTM_cell. Use "x", not "x0" (≈1 line)
        a, _, c = LSTM_cell(inputs = x, initial_state=[a,c])
        
        # Step 2.B: Apply Dense layer to the hidden state output of the LSTM_cell (≈1 line)
        out = densor(a)
        # Step 2.C: Append the prediction "out" to "outputs". out.shape = (None, 90) (≈1 line)
        outputs.append(out)
 
        # Step 2.D: 
        # Select the next value according to "out",
        # Set "x" to be the one-hot representation of the selected value
        # See instructions above.
        x = tf.math.argmax(out,axis=-1)
        x = tf.one_hot(x,n_values)
        # Step 2.E: 
        # Use RepeatVector(1) to convert x into a tensor with shape=(None, 1, 90)
        x = RepeatVector(1)(x)
        
    # Step 3: Create model instance with the correct "inputs" and "outputs" (≈1 line)
    inference_model = Model(inputs=[x0, a0, c0], outputs=outputs)
    
    ### END CODE HERE ###
    
    return inference_model

Run the cell below to define your inference model. This model is hard coded to generate 50 values.

In [57]:
inference_model = music_inference_model(LSTM_cell, densor, Ty = 50)
In [58]:
# UNIT TEST
inference_summary = summary(inference_model) 
comparator(inference_summary, music_inference_model_out)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-58-c395f100af16> in <module>
      1 # UNIT TEST
----> 2 inference_summary = summary(inference_model)
      3 comparator(inference_summary, music_inference_model_out)

~/work/W1A3/test_utils.py in summary(model)
     36     result = []
     37     for layer in model.layers:
---> 38         descriptors = [layer.__class__.__name__, layer.output_shape, layer.count_params()]
     39         if (type(layer) == Conv2D):
     40             descriptors.append(layer.padding)

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py in output_shape(self)
   2190                            'ill-defined for the layer. '
   2191                            'Use `get_output_shape_at(node_index)` '
-> 2192                            'instead.' % self.name)
   2193 
   2194   @property

AttributeError: The layer "lstm" has multiple inbound nodes, with different output shapes. Hence the notion of "output shape" is ill-defined for the layer. Use `get_output_shape_at(node_index)` instead.
In [44]:
# Check the inference model
inference_model.summary()
Model: "functional_11"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_10 (InputLayer)           [(None, 1, 90)]      0                                            
__________________________________________________________________________________________________
c0 (InputLayer)                 [(None, 64)]         0                                            
__________________________________________________________________________________________________
a0 (InputLayer)                 [(None, 64)]         0                                            
__________________________________________________________________________________________________
lstm (LSTM)                     multiple             39680       input_10[0][0]                   
                                                                 c0[0][0]                         
                                                                 a0[0][0]                         
                                                                 repeat_vector_201[0][0]          
                                                                 lstm[234][2]                     
                                                                 lstm[234][0]                     
                                                                 repeat_vector_202[0][0]          
                                                                 lstm[235][2]                     
                                                                 lstm[235][0]                     
                                                                 repeat_vector_203[0][0]          
                                                                 lstm[236][2]                     
                                                                 lstm[236][0]                     
                                                                 repeat_vector_204[0][0]          
                                                                 lstm[237][2]                     
                                                                 lstm[237][0]                     
                                                                 repeat_vector_205[0][0]          
                                                                 lstm[238][2]                     
                                                                 lstm[238][0]                     
                                                                 repeat_vector_206[0][0]          
                                                                 lstm[239][2]                     
                                                                 lstm[239][0]                     
                                                                 repeat_vector_207[0][0]          
                                                                 lstm[240][2]                     
                                                                 lstm[240][0]                     
                                                                 repeat_vector_208[0][0]          
                                                                 lstm[241][2]                     
                                                                 lstm[241][0]                     
                                                                 repeat_vector_209[0][0]          
                                                                 lstm[242][2]                     
                                                                 lstm[242][0]                     
                                                                 repeat_vector_210[0][0]          
                                                                 lstm[243][2]                     
                                                                 lstm[243][0]                     
                                                                 repeat_vector_211[0][0]          
                                                                 lstm[244][2]                     
                                                                 lstm[244][0]                     
                                                                 repeat_vector_212[0][0]          
                                                                 lstm[245][2]                     
                                                                 lstm[245][0]                     
                                                                 repeat_vector_213[0][0]          
                                                                 lstm[246][2]                     
                                                                 lstm[246][0]                     
                                                                 repeat_vector_214[0][0]          
                                                                 lstm[247][2]                     
                                                                 lstm[247][0]                     
                                                                 repeat_vector_215[0][0]          
                                                                 lstm[248][2]                     
                                                                 lstm[248][0]                     
                                                                 repeat_vector_216[0][0]          
                                                                 lstm[249][2]                     
                                                                 lstm[249][0]                     
                                                                 repeat_vector_217[0][0]          
                                                                 lstm[250][2]                     
                                                                 lstm[250][0]                     
                                                                 repeat_vector_218[0][0]          
                                                                 lstm[251][2]                     
                                                                 lstm[251][0]                     
                                                                 repeat_vector_219[0][0]          
                                                                 lstm[252][2]                     
                                                                 lstm[252][0]                     
                                                                 repeat_vector_220[0][0]          
                                                                 lstm[253][2]                     
                                                                 lstm[253][0]                     
                                                                 repeat_vector_221[0][0]          
                                                                 lstm[254][2]                     
                                                                 lstm[254][0]                     
                                                                 repeat_vector_222[0][0]          
                                                                 lstm[255][2]                     
                                                                 lstm[255][0]                     
                                                                 repeat_vector_223[0][0]          
                                                                 lstm[256][2]                     
                                                                 lstm[256][0]                     
                                                                 repeat_vector_224[0][0]          
                                                                 lstm[257][2]                     
                                                                 lstm[257][0]                     
                                                                 repeat_vector_225[0][0]          
                                                                 lstm[258][2]                     
                                                                 lstm[258][0]                     
                                                                 repeat_vector_226[0][0]          
                                                                 lstm[259][2]                     
                                                                 lstm[259][0]                     
                                                                 repeat_vector_227[0][0]          
                                                                 lstm[260][2]                     
                                                                 lstm[260][0]                     
                                                                 repeat_vector_228[0][0]          
                                                                 lstm[261][2]                     
                                                                 lstm[261][0]                     
                                                                 repeat_vector_229[0][0]          
                                                                 lstm[262][2]                     
                                                                 lstm[262][0]                     
                                                                 repeat_vector_230[0][0]          
                                                                 lstm[263][2]                     
                                                                 lstm[263][0]                     
                                                                 repeat_vector_231[0][0]          
                                                                 lstm[264][2]                     
                                                                 lstm[264][0]                     
                                                                 repeat_vector_232[0][0]          
                                                                 lstm[265][2]                     
                                                                 lstm[265][0]                     
                                                                 repeat_vector_233[0][0]          
                                                                 lstm[266][2]                     
                                                                 lstm[266][0]                     
                                                                 repeat_vector_234[0][0]          
                                                                 lstm[267][2]                     
                                                                 lstm[267][0]                     
                                                                 repeat_vector_235[0][0]          
                                                                 lstm[268][2]                     
                                                                 lstm[268][0]                     
                                                                 repeat_vector_236[0][0]          
                                                                 lstm[269][2]                     
                                                                 lstm[269][0]                     
                                                                 repeat_vector_237[0][0]          
                                                                 lstm[270][2]                     
                                                                 lstm[270][0]                     
                                                                 repeat_vector_238[0][0]          
                                                                 lstm[271][2]                     
                                                                 lstm[271][0]                     
                                                                 repeat_vector_239[0][0]          
                                                                 lstm[272][2]                     
                                                                 lstm[272][0]                     
                                                                 repeat_vector_240[0][0]          
                                                                 lstm[273][2]                     
                                                                 lstm[273][0]                     
                                                                 repeat_vector_241[0][0]          
                                                                 lstm[274][2]                     
                                                                 lstm[274][0]                     
                                                                 repeat_vector_242[0][0]          
                                                                 lstm[275][2]                     
                                                                 lstm[275][0]                     
                                                                 repeat_vector_243[0][0]          
                                                                 lstm[276][2]                     
                                                                 lstm[276][0]                     
                                                                 repeat_vector_244[0][0]          
                                                                 lstm[277][2]                     
                                                                 lstm[277][0]                     
                                                                 repeat_vector_245[0][0]          
                                                                 lstm[278][2]                     
                                                                 lstm[278][0]                     
                                                                 repeat_vector_246[0][0]          
                                                                 lstm[279][2]                     
                                                                 lstm[279][0]                     
                                                                 repeat_vector_247[0][0]          
                                                                 lstm[280][2]                     
                                                                 lstm[280][0]                     
                                                                 repeat_vector_248[0][0]          
                                                                 lstm[281][2]                     
                                                                 lstm[281][0]                     
                                                                 repeat_vector_249[0][0]          
                                                                 lstm[282][2]                     
                                                                 lstm[282][0]                     
__________________________________________________________________________________________________
dense (Dense)                   multiple             5850        lstm[234][0]                     
                                                                 lstm[235][0]                     
                                                                 lstm[236][0]                     
                                                                 lstm[237][0]                     
                                                                 lstm[238][0]                     
                                                                 lstm[239][0]                     
                                                                 lstm[240][0]                     
                                                                 lstm[241][0]                     
                                                                 lstm[242][0]                     
                                                                 lstm[243][0]                     
                                                                 lstm[244][0]                     
                                                                 lstm[245][0]                     
                                                                 lstm[246][0]                     
                                                                 lstm[247][0]                     
                                                                 lstm[248][0]                     
                                                                 lstm[249][0]                     
                                                                 lstm[250][0]                     
                                                                 lstm[251][0]                     
                                                                 lstm[252][0]                     
                                                                 lstm[253][0]                     
                                                                 lstm[254][0]                     
                                                                 lstm[255][0]                     
                                                                 lstm[256][0]                     
                                                                 lstm[257][0]                     
                                                                 lstm[258][0]                     
                                                                 lstm[259][0]                     
                                                                 lstm[260][0]                     
                                                                 lstm[261][0]                     
                                                                 lstm[262][0]                     
                                                                 lstm[263][0]                     
                                                                 lstm[264][0]                     
                                                                 lstm[265][0]                     
                                                                 lstm[266][0]                     
                                                                 lstm[267][0]                     
                                                                 lstm[268][0]                     
                                                                 lstm[269][0]                     
                                                                 lstm[270][0]                     
                                                                 lstm[271][0]                     
                                                                 lstm[272][0]                     
                                                                 lstm[273][0]                     
                                                                 lstm[274][0]                     
                                                                 lstm[275][0]                     
                                                                 lstm[276][0]                     
                                                                 lstm[277][0]                     
                                                                 lstm[278][0]                     
                                                                 lstm[279][0]                     
                                                                 lstm[280][0]                     
                                                                 lstm[281][0]                     
                                                                 lstm[282][0]                     
                                                                 lstm[283][0]                     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_204 (TensorF [(None,)]            0           dense[234][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_203 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_204[0][0]     
__________________________________________________________________________________________________
repeat_vector_201 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_203[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_205 (TensorF [(None,)]            0           dense[235][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_204 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_205[0][0]     
__________________________________________________________________________________________________
repeat_vector_202 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_204[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_206 (TensorF [(None,)]            0           dense[236][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_205 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_206[0][0]     
__________________________________________________________________________________________________
repeat_vector_203 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_205[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_207 (TensorF [(None,)]            0           dense[237][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_206 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_207[0][0]     
__________________________________________________________________________________________________
repeat_vector_204 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_206[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_208 (TensorF [(None,)]            0           dense[238][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_207 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_208[0][0]     
__________________________________________________________________________________________________
repeat_vector_205 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_207[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_209 (TensorF [(None,)]            0           dense[239][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_208 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_209[0][0]     
__________________________________________________________________________________________________
repeat_vector_206 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_208[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_210 (TensorF [(None,)]            0           dense[240][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_209 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_210[0][0]     
__________________________________________________________________________________________________
repeat_vector_207 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_209[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_211 (TensorF [(None,)]            0           dense[241][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_210 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_211[0][0]     
__________________________________________________________________________________________________
repeat_vector_208 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_210[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_212 (TensorF [(None,)]            0           dense[242][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_211 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_212[0][0]     
__________________________________________________________________________________________________
repeat_vector_209 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_211[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_213 (TensorF [(None,)]            0           dense[243][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_212 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_213[0][0]     
__________________________________________________________________________________________________
repeat_vector_210 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_212[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_214 (TensorF [(None,)]            0           dense[244][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_213 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_214[0][0]     
__________________________________________________________________________________________________
repeat_vector_211 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_213[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_215 (TensorF [(None,)]            0           dense[245][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_214 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_215[0][0]     
__________________________________________________________________________________________________
repeat_vector_212 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_214[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_216 (TensorF [(None,)]            0           dense[246][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_215 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_216[0][0]     
__________________________________________________________________________________________________
repeat_vector_213 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_215[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_217 (TensorF [(None,)]            0           dense[247][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_216 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_217[0][0]     
__________________________________________________________________________________________________
repeat_vector_214 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_216[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_218 (TensorF [(None,)]            0           dense[248][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_217 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_218[0][0]     
__________________________________________________________________________________________________
repeat_vector_215 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_217[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_219 (TensorF [(None,)]            0           dense[249][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_218 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_219[0][0]     
__________________________________________________________________________________________________
repeat_vector_216 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_218[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_220 (TensorF [(None,)]            0           dense[250][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_219 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_220[0][0]     
__________________________________________________________________________________________________
repeat_vector_217 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_219[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_221 (TensorF [(None,)]            0           dense[251][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_220 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_221[0][0]     
__________________________________________________________________________________________________
repeat_vector_218 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_220[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_222 (TensorF [(None,)]            0           dense[252][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_221 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_222[0][0]     
__________________________________________________________________________________________________
repeat_vector_219 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_221[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_223 (TensorF [(None,)]            0           dense[253][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_222 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_223[0][0]     
__________________________________________________________________________________________________
repeat_vector_220 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_222[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_224 (TensorF [(None,)]            0           dense[254][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_223 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_224[0][0]     
__________________________________________________________________________________________________
repeat_vector_221 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_223[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_225 (TensorF [(None,)]            0           dense[255][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_224 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_225[0][0]     
__________________________________________________________________________________________________
repeat_vector_222 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_224[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_226 (TensorF [(None,)]            0           dense[256][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_225 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_226[0][0]     
__________________________________________________________________________________________________
repeat_vector_223 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_225[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_227 (TensorF [(None,)]            0           dense[257][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_226 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_227[0][0]     
__________________________________________________________________________________________________
repeat_vector_224 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_226[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_228 (TensorF [(None,)]            0           dense[258][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_227 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_228[0][0]     
__________________________________________________________________________________________________
repeat_vector_225 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_227[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_229 (TensorF [(None,)]            0           dense[259][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_228 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_229[0][0]     
__________________________________________________________________________________________________
repeat_vector_226 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_228[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_230 (TensorF [(None,)]            0           dense[260][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_229 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_230[0][0]     
__________________________________________________________________________________________________
repeat_vector_227 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_229[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_231 (TensorF [(None,)]            0           dense[261][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_230 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_231[0][0]     
__________________________________________________________________________________________________
repeat_vector_228 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_230[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_232 (TensorF [(None,)]            0           dense[262][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_231 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_232[0][0]     
__________________________________________________________________________________________________
repeat_vector_229 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_231[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_233 (TensorF [(None,)]            0           dense[263][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_232 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_233[0][0]     
__________________________________________________________________________________________________
repeat_vector_230 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_232[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_234 (TensorF [(None,)]            0           dense[264][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_233 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_234[0][0]     
__________________________________________________________________________________________________
repeat_vector_231 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_233[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_235 (TensorF [(None,)]            0           dense[265][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_234 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_235[0][0]     
__________________________________________________________________________________________________
repeat_vector_232 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_234[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_236 (TensorF [(None,)]            0           dense[266][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_235 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_236[0][0]     
__________________________________________________________________________________________________
repeat_vector_233 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_235[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_237 (TensorF [(None,)]            0           dense[267][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_236 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_237[0][0]     
__________________________________________________________________________________________________
repeat_vector_234 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_236[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_238 (TensorF [(None,)]            0           dense[268][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_237 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_238[0][0]     
__________________________________________________________________________________________________
repeat_vector_235 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_237[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_239 (TensorF [(None,)]            0           dense[269][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_238 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_239[0][0]     
__________________________________________________________________________________________________
repeat_vector_236 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_238[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_240 (TensorF [(None,)]            0           dense[270][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_239 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_240[0][0]     
__________________________________________________________________________________________________
repeat_vector_237 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_239[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_241 (TensorF [(None,)]            0           dense[271][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_240 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_241[0][0]     
__________________________________________________________________________________________________
repeat_vector_238 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_240[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_242 (TensorF [(None,)]            0           dense[272][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_241 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_242[0][0]     
__________________________________________________________________________________________________
repeat_vector_239 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_241[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_243 (TensorF [(None,)]            0           dense[273][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_242 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_243[0][0]     
__________________________________________________________________________________________________
repeat_vector_240 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_242[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_244 (TensorF [(None,)]            0           dense[274][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_243 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_244[0][0]     
__________________________________________________________________________________________________
repeat_vector_241 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_243[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_245 (TensorF [(None,)]            0           dense[275][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_244 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_245[0][0]     
__________________________________________________________________________________________________
repeat_vector_242 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_244[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_246 (TensorF [(None,)]            0           dense[276][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_245 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_246[0][0]     
__________________________________________________________________________________________________
repeat_vector_243 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_245[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_247 (TensorF [(None,)]            0           dense[277][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_246 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_247[0][0]     
__________________________________________________________________________________________________
repeat_vector_244 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_246[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_248 (TensorF [(None,)]            0           dense[278][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_247 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_248[0][0]     
__________________________________________________________________________________________________
repeat_vector_245 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_247[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_249 (TensorF [(None,)]            0           dense[279][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_248 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_249[0][0]     
__________________________________________________________________________________________________
repeat_vector_246 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_248[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_250 (TensorF [(None,)]            0           dense[280][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_249 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_250[0][0]     
__________________________________________________________________________________________________
repeat_vector_247 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_249[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_251 (TensorF [(None,)]            0           dense[281][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_250 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_251[0][0]     
__________________________________________________________________________________________________
repeat_vector_248 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_250[0][0]     
__________________________________________________________________________________________________
tf_op_layer_ArgMax_252 (TensorF [(None,)]            0           dense[282][0]                    
__________________________________________________________________________________________________
tf_op_layer_OneHot_251 (TensorF [(None, 90)]         0           tf_op_layer_ArgMax_252[0][0]     
__________________________________________________________________________________________________
repeat_vector_249 (RepeatVector (None, 1, 90)        0           tf_op_layer_OneHot_251[0][0]     
==================================================================================================
Total params: 45,530
Trainable params: 45,530
Non-trainable params: 0
__________________________________________________________________________________________________

Expected Output

If you scroll to the bottom of the output, you'll see:

Total params: 45,530
Trainable params: 45,530
Non-trainable params: 0

Initialize inference model

The following code creates the zero-valued vectors you will use to initialize x and the LSTM state variables a and c.

In [45]:
x_initializer = np.zeros((1, 1, n_values))
a_initializer = np.zeros((1, n_a))
c_initializer = np.zeros((1, n_a))

Exercise 3 - predict_and_sample

Implement predict_and_sample().

This function takes many arguments, including the inputs x_initializer, a_initializer, and c_initializer.

In order to predict the output corresponding to this input, you'll need to carry out 3 steps:

Step 1:

  • Use your inference model to predict an output given your set of inputs. The output pred should be a list of length $T_y$ where each element is a numpy-array of shape (1, n_values).
    inference_model.predict([input_x_init, hidden_state_init, cell_state_init])
    
    • Choose the appropriate input arguments to predict from the input arguments of this predict_and_sample function.

Step 2:

  • Convert pred into a numpy array of $T_y$ indices.
    • Each index is computed by taking the argmax of an element of the pred list.
    • Use numpy.argmax.
    • Set the axis parameter.
      • Remember that the shape of the prediction is $(m, T_{y}, n_{values})$

Step 3:

  • Convert the indices into their one-hot vector representations.
    • Use to_categorical.
    • Set the num_classes parameter. Note that for grading purposes: you'll need to either:
      • Use a dimension from the given parameters of predict_and_sample() (for example, one of the dimensions of x_initializer has the value for the number of distinct classes).
      • Or just hard code the number of distinct classes (will pass the grader as well).
      • Note that using a global variable such as n_values will not work for grading purposes.
In [46]:
# UNQ_C3 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: predict_and_sample

def predict_and_sample(inference_model, x_initializer = x_initializer, a_initializer = a_initializer, 
                       c_initializer = c_initializer):
    """
    Predicts the next value of values using the inference model.
    
    Arguments:
    inference_model -- Keras model instance for inference time
    x_initializer -- numpy array of shape (1, 1, 90), one-hot vector initializing the values generation
    a_initializer -- numpy array of shape (1, n_a), initializing the hidden state of the LSTM_cell
    c_initializer -- numpy array of shape (1, n_a), initializing the cell state of the LSTM_cel
    
    Returns:
    results -- numpy-array of shape (Ty, 90), matrix of one-hot vectors representing the values generated
    indices -- numpy-array of shape (Ty, 1), matrix of indices representing the values generated
    """
    
    n_values = x_initializer.shape[2]
    
    ### START CODE HERE ###
    # Step 1: Use your inference model to predict an output sequence given x_initializer, a_initializer and c_initializer.
    pred = inference_model.predict([x_initializer, a_initializer, c_initializer])
    # Step 2: Convert "pred" into an np.array() of indices with the maximum probabilities
    indices = np.argmax(pred, axis=-1)
    # Step 3: Convert indices to one-hot vectors, the shape of the results should be (Ty, n_values)
    results = to_categorical(indices, num_classes=78)
    ### END CODE HERE ###
    
    return results, indices
In [47]:
results, indices = predict_and_sample(inference_model, x_initializer, a_initializer, c_initializer)

print("np.argmax(results[12]) =", np.argmax(results[12]))
print("np.argmax(results[17]) =", np.argmax(results[17]))
print("list(indices[12:18]) =", list(indices[12:18]))
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-47-e3fa72710400> in <module>
----> 1 results, indices = predict_and_sample(inference_model, x_initializer, a_initializer, c_initializer)
      2 
      3 print("np.argmax(results[12]) =", np.argmax(results[12]))
      4 print("np.argmax(results[17]) =", np.argmax(results[17]))
      5 print("list(indices[12:18]) =", list(indices[12:18]))

<ipython-input-46-f9fa0d8fe3d8> in predict_and_sample(inference_model, x_initializer, a_initializer, c_initializer)
     26     indices = np.argmax(pred, axis=-1)
     27     # Step 3: Convert indices to one-hot vectors, the shape of the results should be (Ty, n_values)
---> 28     results = to_categorical(indices, num_classes=78)
     29     ### END CODE HERE ###
     30 

/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/utils/np_utils.py in to_categorical(y, num_classes, dtype)
     76   n = y.shape[0]
     77   categorical = np.zeros((n, num_classes), dtype=dtype)
---> 78   categorical[np.arange(n), y] = 1
     79   output_shape = input_shape + (num_classes,)
     80   categorical = np.reshape(categorical, output_shape)

IndexError: index 82 is out of bounds for axis 1 with size 78

Expected (Approximate) Output:

  • Your results may likely differ because Keras' results are not completely predictable.
  • However, if you have trained your LSTM_cell with model.fit() for exactly 100 epochs as described above:
    • You should very likely observe a sequence of indices that are not all identical. Perhaps with the following values:
      **np.argmax(results[12])** = 26
      **np.argmax(results[17])** = 7
      **list(indices[12:18])** = [array([26]), array([18]), array([53]), array([27]), array([40]), array([7])]

3.2 - Generate Music

Finally! You're ready to generate music.

Your RNN generates a sequence of values. The following code generates music by first calling your predict_and_sample() function. These values are then post-processed into musical chords (meaning that multiple values or notes can be played at the same time).

Most computational music algorithms use some post-processing because it's difficult to generate music that sounds good without it. The post-processing does things like clean up the generated audio by making sure the same sound is not repeated too many times, or that two successive notes are not too far from each other in pitch, and so on.

One could argue that a lot of these post-processing steps are hacks; also, a lot of the music generation literature has also focused on hand-crafting post-processors, and a lot of the output quality depends on the quality of the post-processing and not just the quality of the model. But this post-processing does make a huge difference, so you should use it in your implementation as well.

Let's make some music!

Run the following cell to generate music and record it into your out_stream. This can take a couple of minutes.

In [63]:
out_stream = generate_music(inference_model, indices_values, chords)
Predicting new values for different set of chords.
Generated 35 sounds using the predicted values for the set of chords ("1") and after pruning
Generated 35 sounds using the predicted values for the set of chords ("2") and after pruning
Generated 35 sounds using the predicted values for the set of chords ("3") and after pruning
Generated 35 sounds using the predicted values for the set of chords ("4") and after pruning
Generated 35 sounds using the predicted values for the set of chords ("5") and after pruning
Your generated music is saved in output/my_music.midi

Using a basic midi to wav parser you can have a rough idea about the audio clip generated by this model. The parser is very limited.

In [64]:
mid2wav('output/my_music.midi')
IPython.display.Audio('./output/rendered.wav')
Out[64]:

To listen to your music, click File->Open... Then go to "output/" and download "my_music.midi". Either play it on your computer with an application that can read midi files if you have one, or use one of the free online "MIDI to mp3" conversion tools to convert this to mp3.

As a reference, here is a 30 second audio clip generated using this algorithm:

In [65]:
IPython.display.Audio('./data/30s_trained_model.wav')
Out[65]:

Congratulations!

You've completed this assignment, and generated your own jazz solo! The Coltranes would be proud.

By now, you've:

  • Applied an LSTM to a music generation task
  • Generated your own jazz music with deep learning
  • Used the flexible Functional API to create a more complex model

This was a lengthy task. You should be proud of your hard work, and hopefully you have some good music to show for it. Cheers and see you next time!

What you should remember:

  • A sequence model can be used to generate musical values, which are then post-processed into midi music.
  • You can use a fairly similar model for tasks ranging from generating dinosaur names to generating original music, with the only major difference being the input fed to the model.
  • In Keras, sequence generation involves defining layers with shared weights, which are then repeated for the different time steps $1, \ldots, T_x$.

4 - References

The ideas presented in this notebook came primarily from three computational music papers cited below. The implementation here also took significant inspiration and used many components from Ji-Sung Kim's GitHub repository.

Finally, a shoutout to François Germain for valuable feedback.

In [ ]: